Method and device for evaluation of eyelashes

ABSTRACT

A device for automating the process of installing eyelash extensions onto the natural eyelashes of a subject. In some embodiments, the placing of extensions is carried out by a robotic mechanism utilizing computer vision.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/423,000, which was filed on Nov. 16, 2016 and titled “Machine forBeauty Salon”. The entire content of this application is incorporatedherein by reference.

FIELD OF THE INVENTION

The invention relates to the process of applying eyelash extensions.

BACKGROUND OF THE INVENTION

Eyelash extensions are increasing in popularity all over the world.Eyelash extensions are usually differentiated from what is called an“artificial eyelash” or an “artificial eyelash structure” by the factthat they are affixed one to one to a natural eyelash fiber. An“artificial eyelash” is a complete set of eyelash fibers (usually forone eye) that is connected to a backing material (a thin strip at theproximal end of the eyelash fibers), which is affixed to the eyelid.This process is therefore simpler and is provided for home use. Eyelashextensions, however, are laboriously glued, usually with a cyanoacrylateadhesive, to each natural eyelash fiber one at a time by a beautytechnician. Extensions may have branches, such as shown in U.S. Pat. No.8,127,774, and there are some schemes for interlocking with nearbyeyelashes, such as disclosed in U.S. Pat. No. 8,113,218.

When eyelash extensions are applied for the first time, the appointmentcan take a considerable amount of time, lasting up to two hours. Duringan appointment, each eyelash extension must be picked up in the properorientation with tweezers, dipped in adhesive, and then placed againstone of the subject's natural eyelash fibers until adhesion occurs.Because this large amount of labor costs beauty salons money, andbecause the length of time required and cost deters some customers,there have been some labor-saving devices proposed. One such device is adispenser for eyelashes that is held in the hand, disclosed in U.S.Patent Application Publication No. 2014/0261514. There have also beenlabor saving proposals regarding the trays on which the extensions comefrom the factory, such as can be seen in U.S. Pat. No. 8,701,685. Thesetrays are intended to combat the fact that it is not only the adhesionstep of the process which is difficult for humans. Just picking theeyelash extensions up with a pair of tweezers is challenging. Also, ithas been proposed that the handling of adhesive and the step of dippingthe extension into adhesive can be eliminated by providing eachextension with a pre-installed piece of heat shrink tubing which is usedto affix the extension to the natural eyelash fiber.

There is a need, therefore, for a way to more effectively installeyelash extensions, which would reduce both the time and the cost ofdoing so. The invention described here applies to all eyelashextensions, whether branched, interlocked, or otherwise, and to allmethods of adhesion to the natural eyelash, whether by adhesive, heatshrink tubing, or otherwise.

SUMMARY OF THE INVENTION

The invention is a machine and method which aid in the application ofeyelash extensions by a robotic mechanism. In particular, the inventionrelates to the process of isolating and imaging eyelashes for extension.Because eyelashes are tightly packed, and often stuck together oroccluded, automatic extension of eyelashes can require an iterativeprocess of isolating and examining eyelashes. The robotic mechanism iscontrolled by a control system which includes a sensor system that canidentify and locate eyelash fibers (some of which are natural and someof which are artificial).

Additional objects, features and advantages of the invention will becomemore readily apparent from the following detailed description ofpreferred embodiments thereof when taken in conjunction with thedrawings wherein like reference numerals refer to common parts in theseveral views.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows the basic external features of an embodiment of theinvention.

FIG. 2 shows an embodiment with an alternate pose of a subject.

FIG. 3 shows the internal components of an enclosure.

FIG. 4 is a detailed view of a robotic head mechanism.

FIG. 5 shows a typical tray of eyelash extensions as provided by themanufacturer.

FIG. 6 is a detailed view of the robotic head mechanism from analternate angle and with the tilt axis moved.

FIG. 7 is a detailed view of the robotic head mechanism with analternate computer vision system.

FIG. 8 is a view looking downward from the backside of the enclosure.

FIG. 9 is a view looking into a subenclosure.

FIG. 10 shows the robotic head mechanism with an eyelash isolatormechanism.

FIG. 11 shows a view looking down upon the face of a subject during theisolation process.

FIG. 12 shows the same view as FIG. 11 after an eyelash isolatormechanism probe has moved.

FIG. 13 shows a manual eyelash isolation tool.

FIG. 14 shows an alternate embodiment using a SCARA robot.

FIG. 15 shows an alternate embodiment using a six-axis robot.

FIG. 16A shows an alternative robotic head mechanism arrangement usingcurved tweezers.

FIG. 16B is a schematic top view of the operation of the alternativerobotic head mechanism.

FIG. 16C is a schematic top view of the operation of the alternativerobotic head mechanism.

FIG. 16D is a schematic top view of the operation of the alternativerobotic head mechanism.

FIG. 16E is a schematic top view of the operation of the alternativerobotic head mechanism.

FIG. 17A is a top view of an alternative robotic head mechanism usingthe curved tweezers of FIGS. 16A-E.

FIG. 17B is a side view of the alternative robotic head mechanism ofFIG. 17A.

FIG. 18 shows an alternative robotic head mechanism similar to that ofFIGS. 17A and B, including two additional degrees of freedom.

FIG. 19A is a first view of a robotic head mechanism.

FIG. 19B is a second view of the robotic head mechanism of FIG. 19A.

FIG. 19C is a third view of the robotic head mechanism of FIG. 19A.

FIG. 19D is a fourth view of the robotic head mechanism of FIG. 19A.

FIG. 20 is a schematic view of an idealized eyelash and a more realisticeyelash.

FIG. 21A shows two eyelashes.

FIG. 21B shows the two eyelashes of FIG. 21A inadvertently bondedtogether.

FIG. 21C shows the irregular growth pattern resulting from the bondingof FIG. 21B.

FIG. 22 shows natural human eyelashes at great magnification,schematically depicting different types of clustering.

FIG. 23 shows various geometric acceptance criteria for evaluation of asingleton eyelash.

FIG. 24A shows how a stereo camera system can help identify eyelashclusters.

FIG. 24B shows how the stereo camera system of FIG. 24A can helpidentify eyelash clusters.

FIG. 24C shows how the stereo camera system of FIG. 24A can helpidentify eyelash clusters.

FIG. 25A shows a first portion of a method of isolation involvingtweezers and not requiring good a priori knowledge of the eyelashlocation.

FIG. 25B shows a second portion of the method of FIG. 25A.

FIG. 25C shows a third portion of the method of FIG. 25A.

FIG. 25D shows a fourth portion of the method of FIG. 25A.

FIG. 26 depicts tweezers operating in the eyelashes of a subject andhaving fiducial markers clear of the working area.

FIG. 27 is a process diagram for iteratively isolating eyelashes andchecking them against acceptance criteria.

FIG. 28 is a process diagram for evaluating eyelashes based on variousgeometric criteria and parameters.

FIG. 29 is a process diagram for the evaluation of eyelashes using bothgeometric and neural network processes.

FIG. 30A shows a first portion of a method of isolation using twoseparate probes and not requiring good knowledge of eyelash locationuntil the latter steps.

FIG. 30B shows a second portion of the method of FIG. 30A.

FIG. 30C shows a third portion of the method of FIG. 30A.

FIG. 30D shows a fourth portion of the method of FIG. 30A.

FIG. 31 shows how a general location of the natural human eyelash can befound using stereo computer vision.

FIG. 32 shows a cart-type mount for the enclosure.

DETAILED DESCRIPTION OF THE INVENTION

The following description sets forth numerous specific configurations,parameters, and the like. It should be recognized, however, that suchdescription is not intended as a limitation on the scope of the presentinvention but is instead provided as a description of exemplaryembodiments.

In the following description, when the term “eyelash” is used, it ismeant to refer to one or more natural eyelash fibers of a person. Whenthe term “eyelash extension” or “extension” is used, it is meant torefer to an artificial eyelash extension. When the term “fan of theeyelashes” is used, it refers to all the natural eyelashes associatedwith an eye.

Robotic Eyelash Extension

FIG. 1 is an external view of a robotic eyelash extension system whichillustrates the basic external features of an embodiment of the presentinvention. This embodiment has an enclosure 201 which is intended toprotect a subject 301 and the user (the word “subject” will be used todenote the person who is receiving the eyelash extensions, and the word“user” will be used to denote the person who is operating the equipment,usually, but not necessarily a beauty technician). Subject 301 reclineson a chair 302 much like the chair in a dentist office or, moreappropriately, the type of beauty salon chair that can recline as isoften done in order to wash a subject's hair. Enclosure 201 is attachedto the floor, ceiling, or chair by an arm 103 which is used to positionenclosure 201 against the face of subject 301 so that the roboticmechanism inside will have access to the eyelashes of subject 301. Arm103 can take many forms (and can even be just a cart that rollsenclosure 201 above subject 301 and has an easy way to set the distancefrom the floor to enclosure 201), but here the arm is meant to beportrayed as the arm typically used in the dental office to positionvarious tools such as an x-ray device. Arm 103 can include a pantographmechanism (not shown) like the ones used in dental offices to keep thedentist's instrument table level at all times as it is moved easily tovarious positions. Arm 103 can also be a simple swing arm rotating abouta vertically oriented pivot. The system in FIG. 1 also shows a window269 which is comprised of an opening on the underside of enclosure 201that permits the robotic mechanism access to the eyelashes of subject301 when enclosure 201 is in proximity to subject 301. Of course, whileenclosure 201 can protect subject 301 and the user from the roboticmechanism, it cannot protect against parts of the robotic mechanism thatcan protrude from window 269. There are several ways to further ensuresafety. First, experts in robotics will note that it is possible tocreate a robot that is inherently safe through strict process controlsand redundant sensing and processing such as is done in surgicalrobotics. However, it is also possible to provide a layer of protectionplaced on the person such that the robotic mechanism cannot reach anyportion of subject 301 through window 269 except their eyelashes. It iseven possible to design the robotic mechanism to be so weak that itcannot damage the human eye (this is possible because the eyelashextension being applied is extremely light and can be manipulated by avery weak robot). For the following discussion, it will generally beassumed that the robot is inherently safe and can operate in closeproximity to the human subject and user without undue risk, although theembodiments presented here are generally appropriate regardless ofchoice of safety system.

In an alternate embodiment, shown in FIG. 2, subject 301, rather thanreclining, leans forward over an alternate chair 122. In FIG. 2,alternate chair 122 is depicted as a large padded pyramidal form, butchair 122 can just as easily be a forward-facing massage chair or akneeling computer chair. In this embodiment, an alternate enclosure 151can sit on the floor or support surface, with an alternate opening 169on its top side. Of course, this will result in a different orientationof the robotic mechanism, but it is well understood in the art ofrobotics how the orientation of a robot can be adjusted. Advantageous tothis alternate embodiment is that subject 301 can remove themselves fromthe system at any time, where in the previous embodiment, enclosure 201is removed before subject 301 can exit. Furthermore, this embodimentalso places alternate enclosure 151 on the ground or support surfacedirectly, or on short legs, reducing the need for other supportstructure. However, it is disadvantageous that many persons find thisposture less comfortable than the reclined posture of the previousembodiment. Nevertheless, the two embodiments can generally use similarrobotic mechanisms and equipment.

In FIG. 3, a front face 216 and a roof 217 (see FIG. 1) of enclosure 201have been omitted so that the inside of enclosure 201 can be seen. Also,a right side 218 of enclosure 201 has been cut away to add visibility.It can be seen that there is a robotic mechanism 219 inside enclosure201. Mechanism 219 is of a type commonly referred to as a “Cartesianrobot” or an “xyz stage.” It is a combination of linear actuators (andsome rotational axes on the head which will be discussed below) whichmove the mechanism linearly in various directions. In FIG. 3, thedirection of the x-axis is shown by an arrow 220, the direction of they-axis by an arrow 221, and the direction of the z-axis by an arrow 222.In FIG. 3, robotic mechanism 219 has two x-axis actuators 223 which arepowered by an x-axis motor 224 which is connected to x-axis belts 225which move x-axis carts 226 back and forth along the x-axis. The wiringand other details used to implement robotic mechanism 219 have beenomitted for clarity.

Robotic mechanism 219 also has a y-axis actuator 227 which is mounted onx-axis carts 226. Y-axis actuator 227 is powered by a y-axis motor 228which is connected to a y-axis belt 229 which moves a y-axis cart 230back and forth along the y-axis. Robotic mechanism 219 also has a z-axisactuator 231 which is mounted on y-axis cart 230. Z-axis actuator 231 ispowered by a z-axis motor 232 which is connected to a z-axis slide 233by a belt which is not visible but which moves z-axis slide 233 up anddown along the z-axis. A robotic head mechanism 234 is connected toz-axis slide 233.

Robotic head mechanism 234 is shown in FIG. 4 with the other parts ofrobotic mechanism 219 omitted for clarity. In FIG. 4, the orientationterms are labeled as a tilt (or pitch) axis 181, a roll axis 182, and atwist (or yaw) axis 180. The first component of robotic head mechanism234 is mounted to z-axis slide 233 and, in this embodiment, is a twistaxis actuator 235. Actuator 235 uses an internal mechanism and motor torotate the remainder of robotic head mechanism 234 about the z-axis.Below twist axis actuator 235 is a tilt actuator bracket 236 whichconnects the bottom of twist axis actuator 235 to a tilt axis actuator237. Actuator 237 uses an internal mechanism and motor to rotate theremainder of robotic head mechanism 234 about an axis perpendicular tothe z-axis but which rotates with twist axis actuator 235. In thisembodiment, there are five axes which are actuated: x (actuators 223); y(actuator 227); z (actuator 231); twist (actuator 235); and tilt(actuator 237). One skilled in the art will note that in order to get atruly arbitrary position of the end effector, a robot must have at leastsix degrees of freedom. However, in this embodiment, the complication ofan additional axis is avoided by ensuring that the extensions arepresented to robotic mechanism 219 in a generally consistentorientation. This is possible because artificial eyelash extensionsgenerally come in regular rows, such as the extensions shown in a row272 of FIG. 5. Regardless, it is easy to see that a roll axis actuatorcan be added beside tilt axis actuator 237 to provide this axis ifdesired.

FIG. 6 is a detailed view of robotic head mechanism 234 from analternate angle and with the tilt axis moved. A gripping actuator 238 ismounted directly to the output of tilt axis actuator 237 and has asimple mechanism to grip and ungrip tweezers 239. Tweezers 239 have astationary side 258 and a moving side 259. By “stationary”, it is meantthat side 258 of tweezers 239 does not move with respect to the rest ofrobotic head mechanism 234 when gripping actuator 238 is actuated butmoving side 259 does. Gripping actuator 238 can be a simple pneumaticactuator, solenoid-type actuator, electric motor, or any number ofgripping mechanisms commonly used. Of course, any of these types ofactuators can be selectively actuated by a computer system orcontroller. Also mounted to tilt axis actuator bracket 236 (andtherefore not moving with the tilt axis in this embodiment) are a camera240 and LED light arrays 241. These are used by the robotic mechanismcontrol system's controller to illuminate and locate both naturaleyelashes and synthetic eyelash extensions. The control system can takethe form of a controller using computer vision (also sometimes referredto as machine vision) techniques to try to estimate the position andorientation of the natural eyelashes and synthetic eyelash extensions.For example, the robotic mechanism control system can be amicroprocessor running the ROS™ operating system and programmed usingcomputer vision routines from the OpenCV™ library in order to performthe basic functions of processing images from camera 240 and estimatingpositions. Camera 240 is a digital camera in communication with acomputer or controller. One skilled in the art will note that, forbetter capability of locating the natural eyelash fibers and syntheticeyelash extensions in three-dimensional space, two cameras can be usedand substituted for camera 240 in a technique referred to as binocularor stereo computer vision. Such a configuration is shown in FIG. 7, withtwo stereo cameras 277 shown, one on either side of a single light 278.The importance of providing stereo cameras 277 in some embodiments isthat it allows the computer vision system to compute the location andorientation of an object in all six (X, Y, Z, A, B, C) spatialdimensions provided that the object can be suitably recognized in imagesfrom both cameras. In some embodiments, a single camera can be used witha mirror and/or prism in order to produce a pair of stereoscopic imagesonto one camera. Alternatively, a structured light range finder orscanning AM or FM laser range finder can be used to locate the naturaleyelash fibers and synthetic eyelash extensions in three-dimensionalspace. In fact, there are many solutions available today that canaccomplish this task. Likewise, illumination can be provided by manysources other than LED, or illumination can be rendered unnecessary byusing a device such as a structured light range finder to accomplish thetask of determining the position and orientation of the fibers.

Looking at the backside of the machine in FIG. 8 (most internal elementshave been removed for clarity), one can see that there is an access door242 which can be opened to provide access to the inside of enclosure201. Inside enclosure 201, there are target zones where the user canleave consumable materials to be used by the machine. In thisembodiment, there are three rectangular zones indicated on a floor 207of the machine. These are an extension A loading zone 243, an extensionB loading zone 244, and an extension C loading zone 245. Zones 243-245are where trays of eyelash extensions of various lengths and/or shapescan be placed for tweezers 239 to have access. Eyelash extensionstypically come on trays, such as an eyelash extension tray 273 shown inFIG. 5, with various rows of extensions mounted to a sticky backingmaterial. In this embodiment, the user can place one size of extensionin extension A loading zone 243, a tray containing another size inextension B loading zone 244, and a tray containing yet another size inextension C loading zone 245. The control system can have a userinterface (such as a touchscreen mounted to the outside of enclosure201—not shown) which would allow the user to designate which region ofthe eye would use size A, which region would use size B, which regionwould use a mix of A and B, etc. Since robotic mechanism 219 is using avision or other system to locate the extensions, the exact placement ofthe tray in the loading zone can be flexible so as to not cost the usermuch time in placing trays. In some embodiments, the trays can includean identifying tag such as a barcode or RFID tag that can be used toautomatically identify the type of tray.

Also seen from the backside of the machine in FIG. 8 are two round zonesdesignated on floor 207 of enclosure 201. These zones are meant toillustrate possible liquid loading zones: a liquid A loading zone 246and a liquid B loading zone 247. In this embodiment, liquid A loadingzone 246 is shown with an adhesive cup 248 placed in the zone. Adhesivecup 248 is a shallow cup in which the user places a few drops of theadhesive (usually cyanoacrylate, but it can be a special adhesive suchas cyanoacrylate augmented with an anti-bacterial compound such assilver ions) which will be used to bond the synthetic extensions to thenatural eyelash fibers. Liquid B loading zone 247 can be used forvarious other liquids. For example, a primer for the adhesive can beplaced in liquid B loading zone 247 in a receptacle that includes aswab-like tool which tweezers 239 can grab in order to distribute primerto the eyelashes of subject 301. However, in practice, it can be easierto have the user apply primer to the eyelashes of subject 301 beforestarting. In some embodiments, the adhesive can be applied by anautomatic applicator that dispenses adhesive on demand. In someembodiments, the adhesive can be cured at an accelerated rate by use ofan applied accelerant that can be chemical, such as water, or energetic,such as UV light. The type of accelerant depends on the type of adhesiveused.

Also seen in FIG. 8 is a controller 276 of robotic mechanism 219. Ascontrollers for robotic mechanisms are well understood in the art,controller 276 is omitted from other views of robotic mechanism 219 andenclosure 201. However, it is understood that controller 276 represents,without loss of generality, the electronics and computing equipment usedfor the control of robotic mechanism 219. This includes the powerelectronics used for controlling motors 224, 228, and 232 and twist andtilt axis actuators 235 and 237, as well as any other motors andactuators of the robot. Furthermore, controller 276 is configured tocommunicate with camera 240 (or stereo cameras 277 in the event that astereo computer vision system is used) and any sensors used by roboticmechanism 219. Controller 276 further includes a computing system thatcan be comprised, without limitation, of one or more of: amicrocontroller, microcomputer, microprocessor, field programmable gatearray (FPGA), graphics processing unit (GPU), or application specificintegrated circuit (ASIC). Controller 276 includes the software used tocoordinate the motion of robotic mechanism 219 with data received fromthe computer vision system and then to carry out the motions describedduring eyelash isolation and extension placement. In some embodiments,controller 276, together with camera 240 or stereo cameras 277, comprisethe computer vision system. In other embodiments, camera 240 or stereocameras 277 include sufficient processing to comprise the computervision system. In general, the term “computer vision system” is usedhere broadly to mean any sensor or group of sensors configured to imagethe environment in 2D or 3D. The use of a computer vision system tovisually coordinate the motion of a robot and/or an end effector isoften referred to as visual servoing. In such a visual servo system, therobot uses information from the computer vision system to correct itsinternal positional model, which is otherwise typically created throughjoint positional sensors. The advantage is that the joint positionalsensors add error at each joint, with additional error accumulating witheach additional joint of the robot, where the computer vision system isan absolute measurement of output position. In some embodiments, onecomputing system can be used for computer vision and the robotic system,or specialized computing systems in communication can be used for each.

Generally, the device can include a user interface provided to allow theuser to configure robotic mechanism 219 and controller 276 in accordancewith the needs of a particular subject. This interface can allow for thespecification of which extension type to be used in which area of thesubject's natural eyelash. In some embodiments, robotic mechanism 219and controller 276 can be provided with communication to a cloudcomputing platform in order to transmit data, usage statistics,diagnostics, and payment information.

Applying Extensions

This paragraph will summarize the main parts of the process of applyingeyelash extensions using the invention. First, the user will discuss thelook that subject 301 desires and select the type of eyelash extensionsto be used. In this embodiment, the user would then put up to threetrays of different lengths of extensions into loading zones 243-245. Theuser can then use the user interface to indicate in which zones aroundthe eye to use which sizes of extensions (as previously discussedabove). If the invention is not to be used to apply primer to thesubject's eyelashes, the user can do that manually at this time. Theuser can then put a few drops of adhesive into adhesive cup 248, putadhesive cup 248 onto liquid A loading zone 246, and close access door242. The user can then tape down the subject's lower eyelash (as iscurrently done when a beauty technician installs eyelash extensions) andposition enclosure 201. At this time, the user can use the userinterface to start the process. Robotic mechanism 219 will then (bymoving itself to appropriate position using actuators 223, 227, 231,235, and 237) use LED lights 241 and camera 240 to inspect the naturaleyelashes and determine if they are positioned properly. The controlsystem will do this using the output of camera 240 and computer visiontechniques described previously. If the eyelashes are not positionedproperly, robotic mechanism 219 will stop the process and prompt theuser to correct the problem. Robotic mechanism 219 can then start theprocess by “preening” the subject's eyelashes using open tweezers 239,isolating the subject's eyelashes one by one to ensure that they are notstuck together (however, this may not normally be required—in any case,isolating eyelashes will be discussed at length below). In some cases,this isolating is referred to as separating. Robotic mechanism 219 willthen proceed to one of loading zones 243-245, use camera 240 to locatethe next extension in a row of extensions on the tray, and use grippingactuator 238 to grip tweezers 239 around the eyelash extension. It willdo this in the proper orientation by using actuators 223, 227, 231, 235,and 237 to position itself properly with respect to the extension to bepicked up. Robotic mechanism 219 will then proceed to liquid A loadingzone 246. Robotic mechanism 219 will then use tilt axis actuator 237 totilt the extension into a position so that the end to be bonded (fromnow on referred to as the proximal end) is more or less vertical inorientation. The system will then use output from camera 240 to locatethe position of adhesive cup 248 and dip the proximal end of theextension into the cup to apply adhesive. Robotic mechanism 219 willthen move the extension near the subject's natural eyelash. The systemwill then use output from camera 240 to determine the position of thefirst natural eyelash to be bonded to. At this point, robotic mechanism219 can, if needed, separate the first eyelash from the others withtechniques to be described below. When that is done, robotic mechanism219 will use actuators 223, 227, 231, 235, and 237 to properly orientthe extension to the natural eyelash and then to place the extension incontact with the natural eyelash, pausing briefly to let the adhesivebond, and then releasing tweezers 239. Robotic mechanism 219 can thenreturn to the area of the appropriate extension loading zone to repeatthe process with the next extension. As this cycle repeats, the systemmay determine that the next natural eyelash to be extended will be in aregion for which a different extension length is prescribed and willtherefore grab an extension from a different extension loading zone243-245.

In some embodiments, it can be desirable to check that an appropriatequantity of adhesive is applied from adhesive cup 248 before roboticmechanism 219 moves the extension near the subject's natural eyelash.This can be done with output from camera 240, with the systemdetermining the amount of adhesive applied based on the width of theextension and adhesive after application of the adhesive. If too muchadhesive is present, some can be removed, either with a dedicatedmechanism or by brushing the adhesive against a designated adhesive dumplocation. This is similar to the strategy used in manual extension whereexcess adhesive is wiped onto the tape below the eye or onto some othergauze.

In some embodiments, it may be desirable to attach one extension tomultiple eyelashes. Doing so is generally avoided because naturaleyelashes grow at different rates and gluing two together can result ina change in orientation of the extension over time. However, in somecases, it may be desired for specific aesthetic reasons. Therefore, ifinstructed, two or more eyelashes can be intentionally bonded togetherwith a single extension.

In some embodiments, adhesives are not used, but instead a heatshrinkable plastic tube (heat shrink) or band can be used. In suchembodiments, the process proceeds as described, but instead of applyingadhesive, the robotic mechanism hooks the heat shrink band with theextension before threading the extension and heat shrink band over thenatural eyelash. The heat shrink is then activated by a hot air sourceattached to the robotic mechanism.

Nail Polish Application

FIG. 9 shows the left side of enclosure 201 where it can be seen thatthere is an additional access window 249, which does not include a doorin this embodiment. Access window 249 is a window through which asubject's hand can be presented to have his or her nails painted. Accesswindow 249 gives access to a subenclosure 250 which is shown protrudingfrom enclosure floor 207. Subenclosure 250 has its own floor 251.Subenclosure 250 is located below the working area of robotic mechanism219 (which is not visible within enclosure 201 in the view of FIG. 9)such that if robotic mechanism 219 were holding a tool such as a nailpolish applicator in tweezers 239, the nail polish applicator couldtouch a hand lying on floor 251, but the remainder of robotic mechanism219 could not touch the hand. This ensures the safety of a subject inthe event of a malfunction of robotic mechanism 219, since it isunlikely that a person could be injured by being struck by a flexiblenail polish applicator. The system confirms that the subject has notmoved his or her hand into the working area of robotic mechanism 219with a light curtain device 252 mounted at an opening 253 betweensubenclosure 250 and main enclosure 201. Light curtain 252 does not runthe whole length of opening 253 such that the nail polish applicatorheld by tweezers 239 can access the subject's finger nails withoutbreaking light curtain 252, which would stop the device. It can be seen,however, that it would be extremely difficult for a subject to gainaccess to main enclosure 201 with his or her hand without breaking lightcurtain 252 and stopping the device. In practice, the user can placenail polish in standard bottles with standard applicator type lids atliquid A loading zone 246 and liquid B loading zone 247 with theapplicator caps loosened so that they can be grabbed by tweezers 239(tweezers 239 can be created with special features to make this easier).The user can then instruct the subject to lay his or her hand on floor251 of sub-enclosure 250 with his or her nails in the accessible zonefor robotic mechanism 219. Then, robotic mechanism 219 can retract theapplicator from the nail polish bottle, adjust the amount of liquid onthe applicator by running it over the edge of the opening of the nailpolish bottle just as a human would, proceed to opening 253, and applypolish to the subject's finger nails. One skilled in the art will notethat robotic mechanism 219 can apply nail polish top coat over the nailpolish and apply nail polish remover (to touch up nails). It is evenpossible that a textile pad can be provided to robotic mechanism 219 sothat it can remove nail polish after applying nail polish remover allover a nail. However, removing nail polish is much easier for a humanthan applying it, so it may not make sense for robotic mechanism 219 toperform this task.

Isolating Lashes

For embodiments that are designed to produce a very high quality or alonger lasting eyelash extension job, a key part of the process ofapplying eyelash extensions is to isolate one natural eyelash from theothers before a synthetic eyelash extension is applied. The reason forthis is the generally accepted belief in the industry that attachingextensions to groups of eyelashes (though useful for the short term)causes problems as the natural eyelashes grow, because they grow atdifferent rates. Therefore, in some embodiments, it is advantageous toisolate a single eyelash before bonding.

There are many ways that this can be accomplished. FIG. 10 shows anaugmented version of robotic head mechanism 234 which includes aneyelash isolator mechanism 254. Eyelash isolator mechanism 254 is heremounted to stationary side 258 of tweezers 239. By “stationary”, it ismeant that that side 258 of tweezers 239 does not move with respect tothe rest of robotic head mechanism 234 when gripping actuator 238 isactuated but moving side 259 does. It is also important to note that thedistance between sides 258 and 259 of tweezers 239 is shown here muchlarger than would be required to pick up an eyelash extension. This hasbeen shown in order to illustrate a mechanism that can pick up otherobjects, for example, the applicator lid of a common nail polish bottle.In some embodiments, tweezers 239 are much smaller and have a muchsmaller gap between sides 258 and 259. Eyelash isolator mechanism 254has a probe 255 which is intended to contact the natural eyelashes ofsubject 301 and is made of a material that is not compatible with theadhesive used in order to ensure that probe 255 will not bond to theeyelashes of subject 301. In this embodiment, eyelash isolator mechanismprobe 255 is mounted on a spring mechanism 256 which is here illustratedas a four-bar linkage with a spring 257. The intent of spring mechanism256 is to ensure that probe 255 extends slightly below (meaning in thedirection along z-axis 222 towards enclosure floor 207) stationary side258 of tweezers 239 whenever probe 255 is not in contact with a surface.

FIG. 11 shows a view looking down upon enclosure floor 207 and window269 with subject 301's natural upper lid eyelashes 260 protruding intoenclosure 201. Probe 255 is shown in cross section. In practice, probe255 can be brought very close to subject 301's cheek and can evencontact it because spring mechanism 256 does not allow probe 255 togenerate much force against any object it encounters. This wouldtypically be done when robotic mechanism 219 arrives with an eyelashextension 261 in tweezers 239 to which adhesive has just been appliedand is ready to be bonded. In fact, probe 255 can land right on top ofone or two of the natural eyelash fibers. Camera 240 can determine whenprobe 255 has been lowered through natural upper lid eyelash 260. Or,probe 255 can contact the cheek of subject 301, causing eyelash isolatormechanism 254 to deflect a small amount (this can be sensed using asensor on mechanism 254 or using camera 240). In some embodiments, probe255 can include a circuit measuring conductance between subject 301 andprobe 255, thereby determining that contact is made when the conductancegreatly increases. Then, z-axis actuator 231 is stopped, and actuators223, 227, and 235 (perhaps others in some embodiments) are used inconcert to move probe 255 in the direction shown by an arrow 262, movingeyelashes in the process. The length of probe 255 is such that, whiledoing so, there is no risk of eyelash extension 261 accidentallycontacting any natural eyelash fibers (and therefore bonding to thewrong eyelash) because eyelash extension 261 is at a higher location(measured in z-axis 222) than the tip of probe 255. Movement proceedsuntil a target natural eyelash fiber 263 is isolated in a position wherethe bond can be made, as shown in FIG. 12. Tweezers 239 can then bemoved into position to perform the bond, putting the proximal end ofeyelash extension 261 in contact with target natural eyelash fiber 263,and while doing so, eyelash isolator mechanism 254 will deflect more,allowing tweezers 239 to approach without probe 255 moving appreciablywith respect to natural upper lid eyelash 260.

This isolation procedure may not always result in isolation of targetnatural eyelash fiber 263 if target natural eyelash fiber 263 is stuckto an adjacent natural eyelash fiber for some reason. If so, aprocedure, which is often called “preening” in the art, can be used toseparate natural eyelash fiber 263 from others. In preening, probe 255can be brought into natural upper lid eyelash 260 as above. This time,however, probe 255 preferably lands right on top of target naturaleyelash fiber 263 or adjacent to it, and probe 255 preferably lands asclose to the proximal end of target natural eyelash fiber 263 aspossible. When probe 255 has moved into natural upper lid eyelash 260 ortouched subject 301's cheek and eyelash isolator mechanism 254 hasdeflected a small amount, which can be sensed using a sensor onmechanism 254 or using camera 240, z-axis actuator 231 is stopped, andthen actuators 223, 227, and 235 (perhaps others in some embodiments)are used in concert to move probe 255 in the radial direction shown byarrows 264. Again, the length of probe 255 is such that while doing so,there is no risk of eyelash extension 261 accidentally contacting anynatural eyelash fibers (and therefore bonding to the wrong eyelashfiber) because eyelash extension 261 is at a higher location (measuredin z-axis 222) than the tip of probe 255. Movement proceeds until probe255 has exited natural upper lid eyelash 260. This process is repeatedin multiple starting positions, and its intent is to get probe 255between target natural eyelash fiber 263 and the natural eyelash fiberto which it is stuck, therefore pulling them apart as the motionproceeds in a radial direction. If target natural eyelash fiber 263 iswell adhered to an adjacent eyelash, the effect of the preening motioncan be to “pluck” one or both of the stuck eyelashes from the eyelid ofsubject 301 which can cause discomfort and be undesirable. For thatreason, the mounting of eyelash isolator mechanism 254 can beinstrumented in a way to detect force on probe 255 which can tell thesystem to stop the motion. At that point, the system can try again, andif target natural eyelash fiber 263 cannot be isolated from others, theuser can be notified and prompted to fix the problem manually or to skiptarget natural eyelash fiber 263. It is also relevant to note that, asindicated above, the computer vision system can locate both probe 255and potentially tweezers 239 as well. This has several advantagesincluding that the computer vision location can help correct for errorin robotic mechanism 219. Furthermore, the relative position betweenprobe 255 and target natural eyelash fiber 263 is more important thanthe absolute position of either. That is, the device does not need toknow the absolute position of target natural eyelash fiber 263, but itdoes preferably know the relative position of target natural eyelashfiber 263 with respect to probe 255 and eyelash extension 261. Indeed,the relative position of the rest of the human eyelashes and the cheekof subject 301 with respect to these same items are also relevant andcan be recorded by the computer vision system.

Two sensors have been mentioned with respect to eyelash isolatormechanism 254. One is a sensor used to determine that probe 255 has beenpushed upward and therefore spring 257 has been deflected somewhat, andanother is a sensor used to detect lateral force on probe 255. Thedeflection sensor can be implemented in a number of ways. For example,if eyelash isolator mechanism 254 is implemented as a four-bar linkageas shown here, an optical or magnetic encoder can be put on one of thejoints of the linkage to detect motion continuously. If it is onlydesired to detect that movement has started, a flag can be affixed toone of the bars which blocks the beam of an optical sensor when thedesired amount of deflection has been reached. Likewise, a magnetic orinductive proximity sensor can be used to detect the approach of thebars as the mechanism is moved. One skilled in the art of automatedequipment design will note that there are many ways in which this can beaccomplished. In any event, such sensors can communicate these readingsback to controller 276.

Likewise, creating a sensor to sense a lateral load on probe 255 can beeffected in many ways. For example, a strain gauge can be affixed to themounting of eyelash isolator mechanism 254 to produce an outputproportional to the lateral load on probe 255. Or, the mount for eyelashisolator mechanism 254 can be made compliant enough so that eyelashisolator mechanism 254 moves laterally under lateral load appreciably sothat the techniques to measure motion mentioned in the precedingparagraph can be used to detect a load that was out of bounds. Also, itis possible that the torque of the actuator motors used in the motioncan be monitored to detect if mechanism 254 has encountered someresistance. One skilled in the art of automated equipment design willnote that there are many ways in which this can be accomplished.

Related to the preening of target natural eyelash fiber 263 mentionedabove is bulk preening. Bulk preening can be carried out the same way.However, it is possible that both sides of tweezers 239 can be used atthe same time as probe 255 to separate many eyelashes at once (assumingtweezers 239 are not holding an extension during bulk preening).Tweezers 239 can be moved into position, putting probe 255 in contactwith the cheek of subject 301, and then continuing to move towards thecheek of subject 301 until the end of the tweezers 239 is within thefibers of natural upper lid eyelash 260 and just above the cheek ofsubject 301. Then, the process of moving tweezers 239 and probe 255radially outward from the subject's eyelid will proceed as before, whenonly probe 255 was used for preening. In some embodiments, preening canbe achieved with a brush. In this embodiment, probe 255 can be replacedby a brush, or tweezers 239 can pick up a brush.

One skilled in the art of designing automated equipment will note thatthere are many ways to implement a mechanism to accomplish eyelashisolation. For example, a completely independent robotic mechanism fromrobotic mechanism 219 can be used to move a probe like that of probe 255through the fan of the eyelashes. This mechanism can be a simpleswinging mechanism which is attached to enclosure 201, and which moves aprobe in a set path in the direction shown by arrow 262. Alternatively,robotic head mechanism 234 can be replaced with one that includes twosmall arms, one for preening and one for picking up and placingextensions. An example of such an alternative robotic head mechanism isthe robot designed by Virtual Incision Corporation of Lincoln, Nebr.,which includes two small arms each with several degrees of freedom.

One will also note that it is possible that some natural eyelashes canbe isolated from the others by manually isolating a subset of eyelashesbefore the machine is used. To do this, the user can use a manualeyelash isolation tool such as Speed Eye Link™ manufactured by IsiswigCo., Ltd. of South Korea. An eyelash isolation tool 270 is shown in FIG.13. To use this device in conjunction with the robotic system disclosedhere, the user places tool 270 on cheek 210 of subject 301 (see FIG. 1).The user then works a corrugated surface 271 under a portion of theupper natural eyelashes of subject 301. Then, probes can be usedmanually to separate the natural eyelashes so that each corrugationcontains only a single natural eyelash. The user then positionsenclosure 201 over the face of subject 301 so that the robotic systemcan place extensions onto that subset of the natural eyelashes ofsubject 301. This process can then be repeated in order to get moresubsets of natural eyelashes extended, eventually producing a fullhigh-quality eyelash extension job without the robotic system needing toisolate natural eyelashes. In a related embodiment, a comb-typestructure can be used to isolate individual natural eyelashes, similarto the purpose of corrugated surface 271.

Other Robots

The embodiments shown have utilized a Cartesian robot to illustraterobotic mechanism 219. However, one skilled in the art will note thatthe same thing can be accomplished by a SCARA-type robot, a delta robot,or even a six-axis industrial style robot, and in fact many differentmechanisms can be used to create robotic mechanism 219, with some ofthese mechanisms looking more like dedicated automated equipment ratherthan any robot style in particular. The Cartesian robot design shownhere is portrayed only for easy disclosure of the invention.

For example, in another embodiment of the invention, consider FIG. 14,showing a SCARA robot 500 operating over the head of subject 301. SCARArobot 500 includes a base link 501, which is coupled to an arm 502through a first powered joint 503 that allows arm 502 to be controllablymoved about a first powered joint axis 504. Arm 502 is further coupledto a link 510 through a second powered joint 511 that allows link 510 tobe controllably moved about a second powered joint axis 512. Finally, anoutput link 515 is connected to link 510 though a powered joint that isnot shown but is internal to link 510. Output link 515 can becontrollably moved about and along an axis 516. That is, output link 515is able to translate along axis 516 and rotate about axis 516, with bothmotions being controllable. This arrangement of three rotational axes,504, 512, and 516, and a fourth translational axis along axis 516 is aclassic SCARA arrangement. It will be clear to one skilled in the art ofrobotics that this arrangement will provide the ability to position arobotic head mechanism 517 in three dimensions, X, Y, and Z, similar tothe Cartesian robot used above, with the additional ability to rotaterobotic head mechanism 517 about Z-axis 516. Indeed, SCARA and Cartesianrobots are used for various similar industrial applications, with smallconsiderations in performance and cost driving the use of one overanother. Robotic head mechanism 517 can be similar to robotic headmechanism 234 of FIG. 4, including a camera and one or more tiltactuators to allow for fine positioning of tweezers 520. The finepositioning of tweezers 520 then allows for correct alignment of anextension 521 with a target eyelash 522 of subject 301.

Of course, an even more capable robot arm widely used in industrialapplications is a six-axis arm 530, as shown in FIG. 15. Here, a baselink 531 is coupled to robotic head mechanism 517 through a series sixrotational joints that rotate about rotational axes 532-537. Many robotmanufacturers make six-axis robotic arms with geometries generally alongthe lines of those shown in FIG. 15. Indeed, six-axis robotic arms dateto the 1960 s and Victor Scheinman's work on the Stanford Arm (althoughthat design included five rotational and one translational axis). Theuse of two more degrees of freedom in six-axis arm 530, as compared toSCARA robot 500, allows for fine control of the orientation of robotichead mechanism 517, obviating the need for additional tilt actuators. Inthis case, robotic head mechanism 517 can simply include tweezers 520and an actuator to open and close tweezers 520. Of course, a camera andlighting are preferably also provided, as in previous embodiments, andcan be placed on robotic head mechanism 517, as shown previously, or beattached to another link of six-axis arm 530 or to enclosure 201.

In some embodiments, it can be desirable to provide two robots, oneorienting a first set of tweezers for isolating an eyelash and thesecond robot orienting a second set of tweezers for placing anextension. In the case where the two robots are both six axis arms, thearrangement is roughly analogous to a human doing eyelash extension witheach of their two arms controlling a set of tweezers. This embodiment isroughly equivalent to having two copies of the robot shown in FIG. 15with very simple robotic head mechanisms. In some embodiments, therobotic head mechanism of each robot can simply comprise actuatedtweezers. This embodiment is seen to be less preferred simply because ofthe cost and complexity associated with having two robotic arms but isfeasible for performing an extension job.

Other Robotic Head Mechanisms

In another embodiment of the invention, robotic head mechanism 234 isreplaced with a more complex robotic head mechanism. This can be usefulto achieve greater precision in extension placement than would bepossible with the previously discussed embodiments, and therefore allowsgreater speed. Consider FIG. 16A, which shows bent tweezers 600 inperspective, top and side views. Bent tweezers 600 include a left hole605 and a right hole 606. FIG. 16B then shows bent tweezers 600 withleft and right motors 601 and 602 respectively, each of left and rightmotors 601 and 602 having a plunger extending to left and right holes605 and 606. Motors 601 and 602 can be any small linear actuators havingenough force to actuate tweezers (typically in the single digit newtonsof force). Without limitation, this includes voice coil motors, linearservo motor, piezoelectric motors, and rotary motors having gearing thatconverts the rotary motion to linear motion, such as ball screwassemblies, rack and pinion assemblies, or even cam or linkagemechanisms. In other words, the exact type of actuator that allowsmotors 601 and 602 to actuate their plungers is not important as long asmotors 601 and 602 are capable of actuating tweezers 600. Further, it isimportant to note that a proximal tweezer end 607 and the bodies of bothmotors 601 and 602 are assumed to be grounded by structure not shown inthis view.

By actuating neither motor, in FIG. 16B, tweezers 600 remain in theirunactuated, spread configuration, with equal spacing off a neutral line603. By actuating just left motor 601, in FIG. 16C, tweezers 600 closeto the right of neutral line 603 to a small gap, suitable for grasping asmall object such as an eyelash extension. Conversely, by actuating justright motor 602, in FIG. 16D, tweezers 600 close to the left of neutralline 603 to a small gap, suitable for grasping a small object such as aneyelash extension. By actuating both motors 601 and 602, in FIG. 16E,tweezers 600 close just around neutral line 603 to a small gap, suitablefor grasping a small object such as an eyelash extension. Thus, byselectively actuating motors 601 and 602, it is possible to grasp asmall object in any of these positions, and then by transitioningbetween two of these positions by relaxing one motor while pushing withthe other, it is possible to position the small object in a new positionbefore releasing it.

FIGS. 17A and B show a simple robotic head mechanism that makes use ofthis arrangement of paired left and right motors. Bearings and supportsurfaces have been omitted in FIGS. 17A and B to preserve clarity. Here,there are two tweezers, isolation tweezers 600A and placement tweezers600B, each with their own left motors (601A and 601B) and right motors(602A and 602B). Tweezers 600A and 600B are nested one within the other.In FIG. 17A, it can be clearly seen that isolation tweezers 600A areoutside placement tweezers 600B. Left and right motors 601A and 602A ofisolation tweezers 600A are not actuated, allowing isolation tweezers600A to be fully open. Isolation tweezers 600A separate human eyelashes612, while placement tweezers 600B hold an eyelash extension 611. Inorder to hold eyelash extension 611, placement tweezers 600B are closed,with both left and right motors 601B and 602B partially actuated.However, in order to hold extension 611 slightly to the right of center,left motor 601B is more actuated than right motor 602B. It should beclear that very slight motions of extension 611 are possible throughvery small actuation of the motors controlling placement tweezers 600B.This is advantageous in the operation of the isolation and extensionplacement systems because very small motions can be used formanipulation of eyelashes and eyelash extensions that are on the orderof 50-100 microns in diameter. Of course, robotic mechanism 219 canprovide gross motion in the same direction (the direction of motiondescribed here would be roughly along y-direction arrow 221), but thisfine motion can provide desired additional resolution in the finalpositioning (since both robotic mechanism 219 and placement tweezers600B can actuate in either positive and negative directions along arrow221 it should be understood that the important point is that axes of themotion are coaxial even though the sense of the sign could bedifferent).

An additional thrusting motor 610 is further provided to create relativelinear motion between isolation tweezers 600A and placement tweezers600B along the direction of an arrow 613. Again, as the entire robotichead mechanism is mounted to robotic mechanism 219 (or another roboticmechanism such as the SCARA or 6 axis robots discussed above), themotion along arrow 613 is somewhat redundant with x-direction arrow 220in FIG. 3. This means that, through combination of gross movements ofthe robotic mechanism and thrusting motor 610, it is possible to producedifferential thrusting motion along the direction of arrow 613 betweenisolation tweezers 600A and placement tweezers 600B. This differentialthrusting motion allows for isolation tweezers 600A, for example, toremain stationary along the direction of arrow 613, while placementtweezers 600B move along the direction of arrow 613 in order to placeextension 611 on the natural human eyelash. Furthermore, becausethrusting motor 610 need not provide the entire range of motion, it canhave a greater resolution, allowing for very small and precise motions.

If the robotic mechanism that is used has many degrees of freedom, suchas a six-axis atm, this robotic head mechanism can be sufficient for theisolation and placement steps. However, in the case of the Cartesian orSCARA robots, there may not be enough degrees of freedom to orientextension 611 (i.e., to align extension 611 angularly with the naturaleyelash so that they are collinear rather than crossing). In theprevious embodiments, this was solved with twist axis actuator 235 andtilt axis actuator 237, which provided two more angular degrees offreedom. Here, a similar strategy can be used, providing these degreesof freedom to the placement tweezers 600B. That is, it is possible toinsert the twist and tilt actuators between the two sets of tweezersjust as in the case of thrusting motor 610. It is generally sufficientto provide fewer degrees of freedom to isolation tweezers 600A as it canbe seen that there are many orientations of isolation tweezers 600A thatwill provide sufficient separation of the natural human eyelashes topermit access but only a few tightly clustered orientations that aresufficiently collinear to allow the adhesive to bond extension 611 tothe target natural eyelash.

This is illustrated schematically in FIG. 18, which is principallycomprised of the side view of FIG. 17B, with the addition of a rollarrow 615, indicating the direction of roll of placement tweezers 600Babout a roll axis 617, and the addition of a pitch arrow 616, indicatingthe direction of pitch of placement tweezers 600A about a pitch axis618. In FIG. 18, the orientation terms are labeled as pitch (or tilt)axis 181, roll axis 182, and yaw (or twist) axis 180. Although shown onthe rear side of placement tweezers 600B, the center of rotation ofpitch axis 618 can be further toward the tip of tweezers 600B. Ofcourse, there are many ways known in the art to control the pitch androll of a set of tweezers, such as, without loss of generality, electricmotors, brushed motors, brushless motors, gear motors, and steppermotors. Because placement tweezers 600A are generally light (on theorder of 50-100 grams is common for tweezers), the amount of torquerequired is very low, and there are many readily available options.

A fully realized embodiment of such a robotic head mechanism is shown inFIGS. 19A-D, which show different views of a robotic head mechanism 650.In FIG. 19B, the orientation terms are labeled as pitch (or tilt) axis181, roll axis 182, and yaw (or twist) axis 180. Initially, attention isdrawn to a mounting stub 651 which includes a collar clamp 654. Collarclamp 654 is configured to be attached to a robot, for example SCARArobot 500, six-axis robot 530, or robotic mechanism 219 if robotic headmechanism 234 is not used. Robotic head mechanism 650 includes nineactuators, each controlling motion about or along a single axis. Eachactuator and axis shall now be described in turn. It should beunderstood that various bearings, mounting hardware, and wires will notbe described for sake of clarity as such elements for these actuatorsare well understood by one skilled in the art of robotic design. Theactuators are all comprised of linear motors including linear bearings;some actuators are comprised of two linear motors for convenience ofdesign, but the two linear motors simply act in unison as if they wereone larger motor. Without limitation, the linear motors are intended toinclude brushless linear motors, brushed linear motors, voice coilmotors, rotary motors with their output converted by mechanism to belinear, or even piezoelectric motors.

Mounting stub 651 is connected to a distal axes mounting bracket 685 bya distal axes pivot shaft 690, and motion about a distal axes pitch axis652 is controlled by a distal axes pitch actuator 653. Both straightplacement tweezers 672 and straight isolation tweezers 673 are attacheddownstream of the motion about distal axes pitch axis 652. Therefore,distal axes pitch actuator 653 generates a pitch motion on both tweezers672 and 673.

Straight isolation tweezers 673 are mounted directly to distal axesmounting bracket 685 at their tail end by an isolation tweezers mountingpivot 687. Many actuators for additional motions are also mounteddirectly to distal axes mounting bracket 685. Distal axes mountingbracket 685 mounts a placement tweezer thrust actuator 656, a rightisolation tweezer actuator 670, a left isolation tweezer actuator 669,and an auxiliary brush actuator 680 (the latter three items will bedescribed later).

Placement tweezer thrust actuator 656 controls thrust motion along aplacement tweezer thrust axis 655 by pushing on a placement tweezerthrust carriage 688. Because straight isolation tweezers 673 are mountedupstream of motion about placement tweezer thrust axis 655, they are notthrusted by placement tweezer thrust actuator 656. However, straightplacement tweezers 672 are mounted downstream of the motion aboutplacement tweezer thrust axis 655. As a result, placement tweezer thrustactuator 656 produces relative thrust between straight placementtweezers 672 and straight isolation tweezers 673. This is similar to theaction of thrusting motor 610 described above.

Placement tweezer thrust carriage 688 mounts a placement tweezer rollactuator 660 (see FIG. 19D), which pivots a placement tweezer rollcarriage 689 about a placement tweezer roll axis 659, which iscoincident with placement tweezer thrust axis 655. Again, this rollmotion is applied only to straight placement tweezers 672, therebyproducing relative roll between straight placement tweezers 672 andisolation tweezers 673.

Placement tweezer roll carriage 689 mounts a placement tweezer pitchactuator 663. Straight placement tweezers 672 have their pitchcontrolled by placement tweezer pitch actuator 663. To do this,placement tweezer pitch actuator 663 moves a placement tweezers mountingpivot 662 up and down (using the orientation of FIG. 19A to define upand down in this case). Straight placement tweezers 672 pivot about aplacement grip axis 666. Again, this pitch motion is applied only tostraight placement tweezers 672, thereby producing relative pitchbetween straight placement tweezers 672 and straight isolation tweezers673.

Both straight placement tweezers 672 and straight isolation tweezers 673include two sets of motors controlling their respective closing. Forclarity, only one of two prongs of each set of tweezers is shown, but itshould be understood that the tweezers are typical straight tweezershaving two identical prongs. This action is the same as described abovein connection with FIGS. 17A and B, allowing for the closing of thetweezers about a variable point and effectively allowing for a gripperwith lateral control of its grip point. For straight placement tweezers672, this motion occurs along placement grip axis 666 with a leftplacement tweezer actuator 664 and a right placement tweezer actuator665. For straight isolation tweezers 673, this motion occurs along anisolation grip axis 671 with left isolation tweezer actuator 669 andright isolation tweezer actuator 670. Of course, either set of tweezerscan be constructed with a non-straight geometry and can be curvedtweezers as above.

It is important to note that placement grip axis 666 and isolation gripaxis 671 are not parallel as drawn. This is because placement tweezerroll carriage 689 is shown slightly displaced about placement tweezerroll axis 659. Because straight placement tweezers 672 are mounteddownstream of placement tweezer roll carriage 689 and straight isolationtweezers 673 are not, this slight displacement produces the misalignmentof placement grip axis 666 and isolation grip axis 671.

In practice, the position of straight isolation tweezers 673 iscontrolled by any robot attached to collar clamp 654, as well as distalaxes pitch actuator 653, left isolation tweezer actuator 669, and rightisolation tweezer actuator 670. This means straight isolation tweezers673 have two degrees of freedom, plus tweezer grip, in addition to thedegrees of freedom of the robot. Conversely, the position of straightplacement tweezers 672 is controlled by any robot attached to collarclamp 654, as well as distal axes pitch actuator 653, and additionallyby placement tweezer thrust actuator 656, placement tweezer rollactuator 660, placement tweezer pitch actuator 663, left placementtweezer actuator 664 and right placement tweezer actuator 665. Thismeans straight placement tweezers 672 have five degrees of freedom, plustweezer grip, in addition to the degrees of freedom of the robot.Therefore, it should be clear to one skilled in the art of roboticsthat, with a sufficiently dexterous robot such as SCARA robot 500 or sixaxis robot 530, robotic head mechanism 650 possesses plenty of degreesof freedom to isolate a single natural eyelash of the person whileplacing an eyelash extension.

Finally, auxiliary brush actuator 680 controls the motion of anauxiliary brush arm 682 through a rack and pinion assembly 684. Thisassembly allows for the selective deployment of a brush or probe (notshown) attached to auxiliary brush arm 682. By choosing a brush or probethat is long enough to reach past the tips of tweezers 672 and 673, thebrush or probe can be used to clean, brush, or jostle the naturaleyelash of the human when deployed but not interfere with tweezers 672and 673 when retracted.

Eyelash Recognition Issues and Relation to Acceptance Criteria

Having discussed various robotic systems in detail, attention will nowturn to identification of eyelashes using computer vision, and, in thefollowing section, how this interacts with isolation.

In general, it is difficult to use computer vision to identify aneyelash because (1) the eyelashes are relatively small, with diameterson the order of 100 microns (100×10⁻⁶ meters), because (2) the eyelashesmay be overlapped, stuck together, or in front of one another, because(3) the human subject may slightly move, and because (4) the humansubject cannot be subjected to excessive light or radiation. There aremany computer vision techniques known to the art that are intended toidentify very small objects. For example, a surface mount 0402 resistorused in manufacture of printed circuit assemblies is 400×200 microns,and computer vision systems of various types are widely used in industryto assemble and verify correct assembly of printed circuit assemblies.However, these systems often use laser scanners or structured lightarrays that work only when the subject is motionless (violating 3 above)and are often too bright to be used near the eyes of a human (violating4 above). Of course, new technology is constantly being developed inthis area, and these systems may eventually be useable with humaneyelashes. However, in a preferred embodiment, cameras with highmagnification (e.g., using macro lenses) can resolve details easily intothe tens of microns, small enough for a human eyelash to be many pixelswide. Such cameras do not require excessively bright light and can haveshort shutter times of less than 50 milliseconds, preventing motionartifacts from being an issue. However, issue (2) above remains: manyeyelashes may be overlapping, obscured, or stuck together.

FIG. 20 shows a simplified schematic representation of a short sectionof human eyelash that illustrates issue (2) above. Idealized eyelashsection 700 is an idealized version, with uniformly distributedeyelashes. This is not representative of a typical human eyelashsection. Realistic eyelash section 701, on the other hand, is morerepresentative of a typical human eyelash section. Realistic eyelashsection 701 includes parallel doubled eyelash 702 (parallel to the fieldof view), stacked double eyelash 703 (two eyelashes stacked into thefield of view), and crossing eyelash 704. It is clear from thisschematic that it is not simple to determine that an eyelash is singleor clustered. Furthermore, the problem is important in eyelash extensionbecause eyelashes should, in general, not be glued together.

The problem of identifying if an eyelash is a singleton or a cluster(i.e., two or more stacked or crossing eyelashes) is particularlyimportant because a person's eyelashes may not grow at uniform rates.That is, it is widely held in the art of manual eyelash extension thattwo adjacent natural eyelashes may grow at different rates. If theadjacent eyelashes are bonded together during the placement process, asmight happen if the eyelashes are crossing or just so close that theadhesive used on an extension bridges the gap, then it is easy to seethat the three-eyelash group (two natural, growing at different rates,and one artificial, bonded to the first two), may start to bend in thedirection of the slower growing natural eyelash. As this direction iseffectively random, the extension will point in a direction notintended, ruining the aesthetics of the overall eyelash extension. Thisis known to practitioners of manual eyelash extension but poses aparticular problem for robotic eyelash extension because the computervision system should recognize any doubled eyelash.

As an illustration of this problem, consider FIGS. 21A-C. In FIG. 21A,slow growing eyelash 711 and fast-growing eyelash 712 are shown alongwith eyelid edge 710. Here, the entire figure is a blown-up view of justtwo of the eyelashes in FIG. 20. In FIG. 21B, extension 715 has beenimproperly applied to both slow growing eyelash 711 and fast-growingeyelash 712, perhaps because they were too close for the computer visionsystem to distinguish them. Glue 714 is shown schematically as a dropbetween each eyelash and extension 715, which is at a proper extensionangle 713A. In FIG. 21C, some days have passed, and fast-growing eyelash712 has pushed the extension to one side. As a result, improperextension angle 713B is much greater than proper extension angle 713A.

Recognizing that an eyelash is doubled can be achieved by measuring itsdimensions and comparing them to typical measurements for people ingeneral and against the other eyelashes of that client. To do this, onecan take advantage of two properties of natural human eyelashes: 1)disregarding the tapered tip of the eyelash, natural human eyelashestend to be of constant diameter, and 2) disregarding the tapered tip(and therefore short eyelashes), diameters of individual eyelashes onthe same subject tend to be very consistent. The tapered tips ofeyelashes are produced during the first days of the eyelash growth cycleand are the first part of the eyelash to be seen outside of the skinsurface. Eyelashes in this phase will be tapered and short. After thisphase, the eyelash shaft that is increasingly exposed at the root as theexposed eyelash lengthens (either by eyelash growth or by the folliclepushing the eyelash outward) is generally of constant diameter. FIG. 22depicts a series of eyelashes on human eyelid 727. Single eyelash 720 isof generally consistent diameter until near its tip. The diameter of atypical eyelash like single eyelash 720 is around 100 microns. First andsecond eyelash 722 and 723 forming cluster 721 could be seen as a singleeyelash measuring 200 microns wide, but this would easily be recognizedas being suspect. Measuring the diameter of an eyelash can be done bymeasuring its width in pixels with a single camera in the computervision system and then converting the pixel width into a diameter basedon the known camera geometry. Of course, this geometry depends on thedistance from the camera to the eyelash, but this will be approximatelyknown, or can be measured accurately, as will be discussed later,allowing for easy disambiguation in a case such as cluster 721.Furthermore, it can be sufficient to simply observe a number ofeyelashes to establish a “typical” diameter in pixels and flag asclusters any objects of substantially greater width (i.e., it may noteven be necessary to convert into standard units of measure).

Nevertheless, in some cases, the eyelashes may be stacked on top of eachother from the perspective of the camera, rather than being side byside. This is shown in stacked cluster 724, with front eyelash 725obscuring back eyelash 726. Or, indeed, the eyelashes may be stacked atsome angle between side-by-side and stacked, resulting in an apparentprojected width that may be within the normal range. In someembodiments, it is possible to identify that two eyelashes areoverlapping based on the texture of eyelashes (in the same way that wemight identify that two pieces of pipe are overlapping at a human scalebecause we see the texture and shading of light on the pipe change atany point where some overlap is visible). Nevertheless, eyelashes havelittle texture, and it may not always be possible to recognize theirshape.

As a solution to this problem, it is possible to use two cameras, and,based on the apparent disparity between the images, calculate the depthof the object in question. Then, the acceptance criteria can be based onany geometric parameter—the depth and width, or diameter, length, shape,curvature, cross section, and thickness or any other set of dimensionalcriteria applied to the cross section of the eyelash or eyelash pair. Insome cases, the eyelashes may only cross for a small section of theirlength, forming an “X”. In such a case, it can be advantageous to applythe acceptance criteria over the length of each eyelash, so that anisolated crossing can be identified. In the limit, this approach amountsto creating a three-dimensional model of each candidate eyelash andchecking that it meets acceptable geometric ranges for an eyelash. Inthis embodiment, the acceptance criteria become distributed parameters;that is, the parameter being used as a criterion for a suitable eyelashfor extension is not a criterion at a single point but a parameterdistributed along the length of the eyelash. Such an approach will alsohelp in identifying and excluding juvenile eyelashes that may not yet besuitable for extension.

FIG. 23 shows some of the basic criteria of eyelash 769 on eyelid 727.Here, eyelash length is measured along the normal from the eyelid asnormal length 760, or along the length of the eyelash as path length761, or even the curvature of path length 761. Furthermore, eyelashdiameter 762 can be used as a criterion, or so can be the distributeddiameter or cross-sectional area as indicated by distributedcross-sections 765. Indeed, the complete three-dimensional model ofeyelash 769 can be compared as the bases for acceptance criteria. Theintention of showing these criteria is not to limit the scope of theinvention but to illustrate several geometric criteria that can be used.

As a further example of the utility of using two cameras for estimationof eyelash parameters, consider FIGS. 24A-C, which show several eyelashsituations in cross section, with the cross section of each eyelashshown schematically as a circle. Here, FIGS. 24A-C show stacked cluster724, left angled cluster 732, and right-angled cluster 733. Also shownare left camera 730L and right camera 730R. In FIG. 24B, right camera730R cannot see both eyelashes, and in FIG. 24C, left camera 730L cannotsee both eyelashes. However, it is trivial to see that in allarrangements, at least one of camera 730L and 730R can see botheyelashes and therefore ascertain that the eyelash cluster is too deepin the direction generally along arrow 735 to be a single eyelash.Indeed, it is trivial to see that very large angle 737 is greater thanlarge angle 736 which is greater than single angle 738. Since singleangle 738 is what would be expected for a single eyelash, it is evidentthat very large and large angles 737 and 736 are too large to representa single eyelash. However, the major difficulty in applying this resultin practice is how to solve the stereo correspondence problem—how tomatch the objects that appear in the two cameras when many such objectsmay be in the field of view. That is, in the more general situation thanthe one shown in FIGS. 24A-C, where many eyelashes are present, how isit possible to match an eyelash seen with one camera to an eyelash seenwith the other camera so that we may infer that both camera 730L and730R are seeing the same eyelash or eyelash cluster? This is a class ofproblems in stereo computer vision that cannot be solved in general.Typical strategies involve matching key points in the left and rightcameras, but this is not generally possible when the field of viewcontains many identical objects with little distinct texture. Theparticular solution that is advantageous here will be addressed below.

Relation to Isolation

Of course, simply knowing that a pair of eyelashes is crossing is notsufficient. While it is possible to simply avoid any eyelashes that donot meet the criteria for extension (i.e., that are immature ordoubled), doing so would result in voids, ruining the aesthetics of theoverall eyelash extension job, as discussed above. Therefore, it can bedesirable to physically jostle and isolate eyelashes. In the simplestembodiment, this can involve brushing up against an eyelash with an endeffector. Indeed, this can even be achieved by the robotic system usedfor manipulating the extensions, even while it is holding an extension.This has the obvious advantage of simplicity since no additional roboticsystem is needed. In an exemplary embodiment, this approach can becombined with the acceptance criteria discussed above in alternation inorder to first establish that an eyelash is potentially not a singleton,then probe the eyelash with the robotic system, then check it againstthe acceptance criteria again in order to see if the eyelash is nowclearly a singleton.

However, even this solution may be insufficient. While in some cases,two eyelashes may simply be stuck together and probing them with therobotic system can cause them to separate, in other cases, they may bepressed against each other so that simply disturbing them does not causethem to separate, or, if it does, they simply come back together oncethe robotic system is removed. Therefore, in some embodiments, a secondrobotic system, or a second appendage from the same robotic system isused to keep an eyelash isolated. For example, as discussed above, afirst tweezer can be used to perform eyelash extension placement, with asecond tweezer is used for isolation. In the case of the isolationtweezer, it is closed to a very small gap, then pushed through the fanof the eyelashes, then opened (methods for determining the location ofthe natural human eyelash are described below). In some percentage ofthe time, this will result in a single eyelash remaining between theprongs of the second tweezer.

This isolation technique is illustrated in more detail in FIGS. 25A-D.Here, eyelashes 799 and eyelid section 800 are shown in two views, inperspective and from above. In FIG. 25A, tweezers 810 approach eyelidsection 800; tweezers 810 are nearly, but not quite, closed. In FIG.25B, tweezers 810 are inserted into eyelashes 799, with only generalknowledge of the position of each eyelash. In some instances, the twoprongs of tweezers 810 may not successfully straddle an eyelash, but inthis case, they straddle isolated eyelash 811. In FIG. 25C, tweezers 810begin to open, pushing aside other eyelashes, before, finally, in FIG.25D, the tweezers are fully open and isolated eyelash 811 is fullyisolated.

This embodiment—using the robotic mechanism to perform isolation inconjunction with acceptance criteria—has a secondary advantage. Evenwith the best stereo computer vision systems that use apparentdifferences in a pair of images of the same object, it can be difficultto match multiple objects in order to create the stereo measurements, asdiscussed above. This is particularly true when there are multiple setsof identical objects, such as an insect screen, bars on a window, orhuman eyelashes. The problem is that in absence of other information, itcan be difficult to determine which object that is present in a firstimage is the same object that is present in a second image. Sometimesother information is available, for example the edge of the window inthe example of bars on a window. If the same window edge is visible inboth images, it is possible to count from that reference to each bar andmatch corresponding bars. With, however, over 100 eyelashes on a typicalhuman eye, this strategy is difficult. Another option is to providefiducial markers placed on the person to provide references to thecomputer vision system, but this may not have enough resolution to befoolproof, and the markers will not be the same depth from the camera asthe eyelashes, limiting their utility. This is where isolation driven bythe robotic mechanism can be useful. Once the isolation tweezers haveopened (as in FIG. 25D, above), there is a much lower effective lineardensity of eyelashes between the prongs, and, in most cases, an easilycountable number. Furthermore, the prongs of the tweezers can easily beidentified, and fiducial markers can trivially be added to them,allowing for a known marker location in the midst of the isolatedeyelash(es). In some embodiments, the tweezers can include fiducialmarkers well clear of the eyelash area, allowing for easy identificationof the tweezers, and then the tip location of the tweezers can beinferred based on the fiducial position and tweezer geometry. Isolatingone or several eyelashes between the tweezers allows for effective androbust stereo matching and therefore accurate estimation of the geometryof the eyelash. And, accurate estimation of the geometry of the eyelashfurther allows for more accurate application of the acceptance criteriadiscussed above.

As an example of tweezers having fiducial markers outside of the regionof the eyelashes, consider FIG. 26. Here, tweezers 950 extend well abovethe eyelash of the subject and contains three fiducial makers 952.Because fiducial markers 952 are in an area outside of the eyelashregion, generally indicated by dashed line 954, there is no difficultyin identifying the orientation of tweezers 950, and, based on apredefined geometric model of the tweezers, inferring the position oftweezer tips 956. In some embodiments, such as the curved tweezersabove, this model will be three dimensional, and the tip location intothe page will be estimated from the tweezer orientation as well.

As a summary of the interlocking imaging and isolation techniquesdiscussed here, consider FIG. 27. Initially, on positioning step 740,the subject is positioned into the robotic and computer vision system.Then the eye area is located in eye locating step 741. In someembodiments, this can be achieved automatically by the computer visionsystem recognizing the shape of a closed human eye. However, in someembodiments, it is preferable for a human operator of the robotic andcomputer vision system to manually align the head of the subject and/orthe robotic system so that the eye of the subject is in the field ofview of the camera. This can be used as an extra safety step to ensurethat the computer vision system is looking at the expected type ofobject.

Next, in imaging step 743, the computer vision system images the fan ofthe eyelashes. In some embodiments, this can be the entire fan of theeyelashes for an eye of the human, but in other embodiments, it can be asmaller region that is only a subset of the entire fan of the eyelashes.Using the data from the computer vision system, the computer willcalculate the position of the fan of the eyelashes with respect to therobotic system. Then the computer will choose a region within the fan ofthe eyelashes for isolation and extension placement in choose regionstep 745. Then, the computer will instruct the robotic system to attemptan isolation of an eyelash in attempt isolation step 746. This maneuvercan be performed open loop (i.e., without use of the computer visionsystem) or closed loop (in which case the area is imaged multiple timeswithin attempt isolation step 746 while the isolation is beingperformed). After this step, the computer vision system will image atleast the target isolation area in post isolation imaging step 747, andthen check whether the eyelash isolation was successful in check step749. This can be achieved though the various parameters and acceptancecriteria discussed above, including distributed parameters along thelength of the eyelash.

If the eyelash is determined to have been successfully isolated, thecomputer will instruct the robot to perform the placement routine inplacement step 751. Because the eyelash was carefully imaged in postisolation imaging step 747 and many geometric characteristics of theeyelash were established in check step 749, the computer can feeddetailed information about the eyelash geometry to the robotic system.In at least a preferred embodiment, this geometric characteristiccomprises a three-dimensional model of the eyelash. Once placement step751 is complete, the computer will instruct the robotic system toretract the isolation robot in retraction step 750. This is alsoperformed in the event that the eyelash isolation was not successful instep 749. After the isolation robot is successful, the computer canreturn along path “A” to step 746 or path “B” to imaging step 743. Inthe preferred embodiment, the computer heuristically chooses one pathover the other at a certain interval. For example, it can be desirableto attempt isolation a number of times in one place along the fan of theeyelashes, choosing path “A” repeatedly, but then choose path “B” if theisolation is, or becomes, unsuccessful. By choosing path “B”, the systemcan reset (in a manner of speaking), reimage a larger area of the fan ofthe eyelashes, and choose a new area in which to work.

In a variation on this process, the geometry of an individual eyelash ofa specific person can be evaluated in order to better check the geometryof each isolated eyelash or cluster. For example, consider the flowchart of FIG. 28. Dashed box 922 indicates the process of evaluating thetypical geometry, which might only be performed once during a session,while the portion of the flow chart outside dashed box 922 is performedon every isolation and placement cycle. Starting on imaging step 920,the entire fan the eyelashes, or a large portion of the fan of theeyelashes, is imaged. Then, naturally occurring singleton eyelashes areidentified in natural isolation step 921. This can be achieved because,typically, at least some eyelashes will be isolated, and it is simple todetermine that they are isolated by looking for highly isolated eyelashtips. In some embodiments, and especially for specific subjects where anisolated eyelash cannot be automatically identified with certainty, thesystem can prompt the user (that is, the person operating the system,not subject 301) to identify isolated eyelashes visually in useridentification step 927. Once known good individual eyelashes areestablished, the geometric parameter and distributed parameters areestablished in estimate geometric parameters step 923. As a final check,these parameters are then stored and compared against known good rangesin verification step 924, and any deviation from these ranges isindicated as an error to the user in error step 926. The user can thenchoose to reestablish the parameters with greater visual identification.Once the parameters are established, the parameters are stored in storestep 925. This storage can be local to the individual system, or it caninclude storage on a remote computer or cloud storage system with areference to that subject so that the parameters can be used in futuresessions with other systems.

With further reference to FIG. 28, placement and isolation steps 931 and945 refer to the previously discussed steps of placing an extension on acurrently isolated eyelash and then isolating a new eyelash. Once thenext eyelash is isolated in step 945, the isolated region is imaged inimage isolation region step 928, which further includes calculating thegeometric parameters from the image. Then in step 929, the newlycalculated geometric parameters are compared to the stored parameterswhich were either generated at the start of that session or recalledfrom local or remote memory. Then placement step 931 can proceed if thenewly calculated parameters are within acceptable ranges (i.e., theisolated eyelash is appropriate for extension), or a new isolation canbe attempted if they are not. In some embodiments, as more eyelashes aresuccessfully isolated, the stored parameters can be adapted and refinedbased on the additional data in adaptation step 930.

In another embodiment, shown in FIG. 29, neural networks, a type ofmachine learning, are used. Neural networks can be trained to recognizevarious types of objects in various images (among many otherapplications). Initially, this is shown in neural network initializationin box 932, which is performed in advance. First, a training set shouldbe gathered and manually labeled, in training set step 933. This can berather painstaking as various orientations and colors of eyelashesshould be imaged and manually labeled. However, the problem is somewhatsimplified by consistent lighting and textures in the controlledenvironment of the system. The set of images used should include bothnon-singleton eyelashes (such as images of crossed eyelashes and stackedeyelashes) as well as properly isolated eyelashes, with both setsproperly labeled. Next is training step 934. During this step, theneural network is trained on the labeled data, and tested against areserved portion of that data. Then, this resultant neural network canbe loaded for use into the software of any number of systems.

One skilled in the art will note that this is not the only way to traina neural network, and in fact, methods for unsupervised learning (thatwhich would not require manual labeling of data) are rapidly developingat this time. For example, rather than create real images of isolated,crossed, and stacked eyelashes, a deep generator network can be used tocreate synthetic images that would better train the neural network.Also, one skilled in the art will note that there are many types ofneural networks, such as deep convolutional neural networks and others,that can be taught to recognize objects. Any or all of these processesand types of neural networks can be used in place of box 932.

After the neural network is trained, this neural network can be used inpractice. First is isolation step 945, which refers to the previouslydiscussed steps isolating a new eyelash. Once the eyelash is isolated,the isolated region is imaged in image isolation region step 936, whichfurther includes calculating the geometric parameters from the image.The geometric parameters are then evaluated in evaluation step 937 as inthe previous embodiment. In parallel, raw or conditioned image data isinput into the neural network, in neural network evaluation step 939.Both the geometric methods and neural network methods output a yes/nosignal indicating whether an isolated eyelash is present. Voting step940 looks for agreement between the techniques, requiring that they bothvote yes. Such a scheme helps to ensure that the isolated eyelash isindeed isolated. In some embodiments, rather than producing a simpleyes/no signal, one or both of the methods produces a score, and votingstep 940 heuristically evaluates the two scores. If voting step 940returns that the eyelash is isolated (or is a cluster of the desiredsize), then placement can be attempted in placement step 943. If votingstep 940 returns that the eyelash is not isolated, then isolation isreattempted in isolation step 945.

It is important to note that the neural network is shown here as anadditional check on the geometric techniques previously described. Thismight make sense because currently the output of neural networks is veryunpredictable when they are input images that for some reason differappreciably from those on which the neural network was trained. However,it is possible that a neural network can be created that would inspireenough confidence to be used on its own. In that case, a system likethat shown in FIG. 29 is used, however, evaluation step 937 is notperformed, and the output of step 939 is solely used to determinewhether to attempt placement.

Alternate Isolation Techniques

In yet another embodiment, the isolation can be performed by twoindependent probes rather than by a pair of tweezers. FIGS. 30A-D showhuman eyelid section 800 with associated eyelashes 799 from above. InFIG. 30A, first probe 801, which would be part of a robotic system andis shown in cross section as a circle, is pushed in between two of theeyelashes. In general, this can be done in any of the middle regions ofthe fan of the eyelashes, and the robotic system driving first probe 801need not know the orientation of associated eyelashes 799. The roboticsystem need not know the orientation of associated eyelashes 799 becauseprobe 801 is of small diameter and any individual eyelash will slide toone side or the other of it as it is pushed generally in direction 798.In FIG. 30B, first probe 801 has approached close to eyelid section 800along first direction 798, stopped, and then been moved generally alongsecond direction 797. This has allowed the upper portion of associatedeyelashes 799 to be moved up, creating clear space 802 above isolatedeyelash 803.

Next, isolated eyelash 803 is examined by the computer vision system,which uses stereo vision as described above to estimate the position ofisolated eyelash 803. This is possible because it is now simple for thecomputer vision system to solve the stereo correspondence problembecause the position of isolated eyelash 803 will be clear in both ofthe stereo cameras. That is, it will be simple to match isolated eyelash803 between the cameras. Furthermore, because the view of isolatedeyelash 803 is generally not occluded on at least one side, the computervision system will be better able to estimate its geometry. Because ofthis, second probe 804 can be accurately positioned just below the tipof isolated eyelash 803. Then, by moving second probe 804 along firstdirection 798, second probe 804 can be reliably inserted betweenisolated eyelash 803 and its lower neighbors. Then, by moving secondprobe 804 generally opposite second direction 797, the eyelashes belowisolated eyelash 803 are moved down, and clear space 802 grows toinclude both sides of isolated eyelash 803, which is now truly isolated.It should be noted that either this method of isolation or the tweezermethod of isolation discussed above can be used in conjunction with theprocedures shown in FIG. 27. That is, FIG. 27 is agnostic to the type ofisolation strategy used.

It should be noted that both isolation strategies have a similarunderlying geometry of two probes closely spaced isolating eyelashes—ina sense, tweezers are simply two joined probes. It also should be notedthat the direction 798 can be modified to perform essentially the sameprocedure in a different way. For example, in FIG. 30A, if first probe801 were above the eyelashes 799 with height being measured on an axiswhich points out of the page (not shown), then probe 801 can proceed indirection 798 without entering the field of eyelashes 799 until itreached a point near eyelid 800 at which time it can be plunged intoeyelashes 799 by moving into the page. Such a strategy can produce aslightly different result than simply moving into the eyelashes indirection 798 (e.g., such a strategy can be helpful in the case of twoeyelashes that cross near their tips).

From this point, the procedure can proceed as described above in theflow chart of FIG. 27. Because isolated eyelash 803 is now trulyisolated, it can be inspected to ensure that it meets the criteria andparameters of a single isolated eyelash. If it is found to be doubled,it is possible to then repeat just the latter steps with second probe804 to see if it is possible to dislodge the doubled eyelashes. Thisembodiment is advantageous because it will produce an isolated eyelasheven in very dense eyelashes where it may not be possible to simply pushtweezers through the fan of the eyelashes in order to isolate eyelashes,yet it still does not require that the position of the eyelashes beindividually known a priori.

In yet another variation, it is possible to accomplish isolation bybrushing the eyelash region with a brush much like those which arereferred to as mascara brushes. Furthermore, there are many other waysthat fibers can be isolated from surrounding fibers that can be appliedhere.

Determining the Location of the Fan of the Eyelashes

In many of the above embodiments, even if the position of every eyelashis not known, it is helpful to know the approximate position of the fanof the eyelashes. That is, it is helpful to know the approximategeometric position of the collection natural eyelashes so that, forexample, tweezers 810 or first and second probe 801 and 804 can beinserted through the eyelashes and stop without relying on a forcemeasurement when they touch subject 301. This can be achieved through adifferent solution of the stereo correspondence problem than above. Inthis embodiment, individual eyelashes are ignored by applying bandpassfilter tuned to the coarse structure formed by groups of neighboringeyelashes rather than individual eyelashes. In one embodiment, thebandpass filtered images are binary sign of Laplacian of Gaussianfiltered images. Signed Laplacian of Gaussian filters are wellunderstood in the art of computer vision as a way to filter images.However, in this embodiment, the bandpass filter is tuned to besensitive to features on the order of 5 to 20 eyelash diameters, whichtends to find clusters of eyelashes. Once images from both the left andright cameras are filtered, an area correlation is performed to find themeasured disparity that best matches the images. In practice, thistechnique has been found to accurately measure the position andorientation of the fan of the eyelashes.

This process can be elucidated with a simple schematic process. ConsiderFIG. 31, which shows rectilinear eyelash 900. It should be understoodthat this process applies just as well to eyelashes that are notperfectly rectilinear and that the eyelash is shown as rectilineareyelash 900 simply for clarity of illustration. Rectilinear eyelash 900is imaged with both right and left cameras, producing right raw image902R and left raw image 902L. Note that right and left raw images 902Rand 902L capture slightly different portions of rectilinear eyelash 900.Next, the right and left raw images 902R and 902L are filtered with aGaussian filter, producing right and left blurred images 904R and 904L.Then a Laplacian operator is applied, which tends to emphasize edgesfound in the input images, resulting in right and left edge images 906Rand 906L. Then, a correlation technique is applied, finding right andleft matched portions 908R and 908L. By measuring the pixel displacementof each of right and left matched portions 908R and 908L, right and leftimage displacements 910R and 910L are produced. From the difference ofthese measurements, apparent stereo disparity 911 is produced, it is astraightforward question of trigonometry to then derive the displacementbetween the cameras and rectilinear eyelash 900 based on apparent stereodisparity 911 given the relative spacing and orientation of the rightand left cameras (which is generally known in advance). In someembodiments, many clusters of eyelashes over the entire natural fan ofthe eyelashes are measured, producing a number of stereo disparitymeasurements. In some embodiments, these can simply be averaged in orderto produce a single displacement to the natural human eyelash, in othersa higher order model representing the curvature of the eyelash can beproduced.

Other Variations on Isolation

For aesthetic reasons, it is sometimes desired to attach one eyelashextension to more than one eyelash, despite the issues associated withdifferential growth rates. In this case, the robot can be configured toglue some extensions specifically in clusters. However, the previousdiscussion on identifying single and grouped eyelashes still applies,since the device should identify appropriately doubled or groupedeyelashes for such a process. And, furthermore, some groups will beunsuitable for grouped extensions (e.g., if there are too many eyelashesin the group or if the constituent eyelashes are unacceptably crossing),and it will be desirable to identify these situations. Of course, thesame types of criteria and parameters discussed above can apply to thesesituations as well, with different thresholds than for individualeyelashes.

In some cases, it can be possible to know the position of many of theeyelashes in the natural fan of the eyelashes. This can be the case ifthe person has sparse eyelashes, if the camera resolution is highrelative to the number of eyelashes, or if fiducial references existthat provide an accurate way to solve the stereo correspondence problem.Indeed, it has been found that examining the distal ends of theeyelashes is the most accurate way to identify individual eyelashpositions because there is good contrast and most eyelashes terminatesingly even if they are proximally crossing or occluded. This techniquecan also be used to count the total number of eyelashes present. Still,even in this case, it can be difficult to know the position of everyeyelash exactly. If a single eyelash is thought to be in a knownlocation, however, either tweezers or a double probe as described abovecan be positioned exactly around this single eyelash and then used toisolate that eyelash. Doing so can simplify the isolation process andspeed the isolation process. Because the isolated eyelash can then beaccurately judged against acceptance criteria and parameters, it willnot be catastrophic if the isolated eyelash turns out to have beenincorrectly identified.

It can therefore be understood, from the foregoing examples, that it isof great importance to be able to determine if an isolated eyelash isindeed a singleton appropriate for extension, regardless of how much maybe known about all the human eyelashes present. It is this interlockinginteraction between isolation and determination if an eyelash is asingleton that is important to many embodiments of automatic eyelashextension.

It is important to note that while the previous discussion assumes astereo camera pair for measurement of depth, there are othertechnologies that are available, such as lidar and laser scanning, thatcan produce depth data. Furthermore, in some embodiments, the camera ismounted to the robotic system and can be moved, simulating a stereocamera pair by changing its orientation. In some embodiments, the cameracan be moved from a top view of an eyelash to a side view of theeyelash. But whatever technique is used, it is advantageous to isolatethe eyelash in order to improve its measurement.

Other Options

FIG. 32 shows a cart-based approach for mounting enclosure 201. In FIG.32, instead of being mounted on arm 103, enclosure 201 is mounted oncart mechanism 265 which rolls on casters 266. Cart mechanism 265includes a telescoping mechanism 267 on each leg (note that casters 266and telescoping mechanism 267 are only labeled on one leg for claritybut are provided on each leg). Telescoping mechanisms 267 can be loadedwith constant (or near constant) load springs which would provide enoughupward force to cancel the weight of enclosure 201. The point of this isto make it very easy for the user to adjust the height of enclosure 201properly. Once the proper height is attained, a lock 306 can be used tolock the telescoping mechanism 267 in place. Locks on the caster wheelscan be provided as well. One skilled in the art will recognize thatthere are many ways to implement such a cart.

Some embodiments can take advantage of tape feeds to feed the extensionswhere extensions are mounted to tape which can be fed to the machine byreels such as in the typical machine used to install components ontocircuit boards. In other embodiments, a continuous fiber can be fed tothe device head which would be made into extensions of various lengthsand curvatures during the process. The fiber can be fed to an endeffector which would have a small tube or pinch rollers at its end wherethe fiber would be presented to the natural eyelash fiber. An automaticadhesive distribution device can then add a drop of adhesive. The systemwould then wait for the bond to take place, then feed an appropriatelength of fiber out, and then cut the fiber to the appropriate lengthwith an automatic cutting device. One can imagine in FIG. 10 thatstationary tweezer side 258 can be a fiber feeder and moving side 259can be the cutting mechanism. Those skilled in the art will note thatthis is much like the operation of a modern wire bonder. In fact, thefiber can be given various curvatures and tapers as it is fed out to getnatural looks and styles. This can be accomplished by a system ofrollers that may or may not use heat in addition to pressure to form thefed fiber into the desired shape.

Since robotic mechanism 219 is intended to move with high accelerations,it may cause undesirable motion of the enclosure 201. For this reason,it can be advisable to outfit some of the axes of motion (namely x and ywhich move the most) with counterweights. To explain how this is done,the y-axis will be discussed. Y-axis actuator 227 can be augmented witha counterweight by adding an additional cart (the y-axis counterweightcart) on the opposite side of y-axis actuator 227 from cart 230. Thiscart can be connected to the opposite side of the loop which y-axis belt229 makes between the pulleys. For this reason, the y-axis counterweightcart would move whenever y-axis cart 230 moves, but in the oppositedirection. If a weight was added to the y-axis counterweight cart whichwas of similar weight as everything attached to y-axis cart 230, thenthe loads on the system caused by accelerating the mass of thecomponents traveling on y-axis cart 230 would tend to be canceled. Asimilar strategy can be employed on the x-axis as well.

In some embodiments, the inside of enclosure 201 is climate controlledto preserve the adhesive in cup 248. This avoids a classic problem ofeyelash extension salons which is that the operators usually prefer tokeep the salon at an ambient temperature which is too low for thesubjects to be truly comfortable. In some embodiments, the climate localto the extension being placed is controlled so that the adhesive can bemore rapidly cured. For example, the humidity can be momentarily raisedafter the extension is placed in order to accelerate the curing of theadhesive. It can be seen that a light mist of water can raise thehumidity nearly instantly, and a small fan can later blow in dry roomair to reduce the humidity. Such a system can preserve the workabilityof the adhesive and yet accelerate its curing, leading to a fasterprocedure time. In other embodiments, an accelerant is used other thanwater to accelerate the curing of the adhesive upon placement.

In some embodiments, the adhesive is provided in a dispenser, such as abottle or a syringe that is actuated by the controller. In thisembodiment, clean adhesive is supplied from the dispenser as needed. Forexample, the dispenser can be located in one of liquid loading zones 246and 247, periodically dispensing adhesive into a cup like adhesive cup248.

In some embodiments, the machine controller is connected to the internetso that it can charge a monetary sum to the machine operator/owner foreach procedure. The system can use the internet connection to verifythat the operator/owner has their account in good standing, and if so,it can perform the eyelash extension session and debit their account foreach procedure. Or, it could ensure that the owner/operator has paid inaccordance with some monthly, quarterly, or annual subscription for theright to operate the machine.

While some of the discussion above has focused on the use of tweezers orprobes, it should be recognized that other “ends” can be used, ifdesired. For purposes of the present invention, the “end” or “ends” of arobot are the most distal point or points of the robot.

In some embodiments, the same device may be used for extending eyebrowsrather than eyelashes, as eyebrows have rather similar properties toeyelashes.

Based on the above, it should be readily apparent that the presentinvention provides a way to more effectively install eyelash extensions,which reduces both the time and the cost of doing so. Although describedwith reference to preferred embodiments, it should be readily understoodthat various changes or modifications could be made to the inventionwithout departing from the spirit thereof. In general, the invention isonly intended to be limited by the scope of the following claims.

The invention claimed is:
 1. A method of determining whether a naturaleyelash of a plurality of natural eyelashes of a subject is suitable forextension using at least one camera configured to observe the pluralityof natural eyelashes, a robotic mechanism configured to probe theplurality of natural eyelashes and a controller configured to receivedata from the at least one camera and communicate with the roboticmechanism, the method comprising: measuring a geometric parameter of thenatural eyelash with the at least one camera; comparing the geometricparameter against an acceptable value; and determining, using thecontroller, whether the natural eyelash is suitable for application ofan artificial eyelash extension based on the comparison.
 2. The methodof claim 1, further comprising: determining, based on the comparison,whether the natural eyelash is comprised of multiple natural eyelashes;and if the natural eyelash is comprised of multiple natural eyelashes,determining that the natural eyelash is not suitable for extension. 3.The method of claim 1, wherein the at least one camera includes a firstcamera and a second camera, the method further comprising: imaging thenatural eyelash with the first camera to produce a first image; andimaging the natural eyelash with the second camera to produce a secondimage, wherein measuring the geometric parameter includes measuring adepth of the natural eyelash based on an apparent disparity between thefirst and second images, and comparing the geometric parameter includescomparing the depth against an acceptable depth value.
 4. The method ofclaim 1, wherein measuring the geometric parameter includes measuringthe geometric parameter at multiple points along the natural eyelash togenerate a distributed geometric parameter, and comparing the geometricparameter includes comparing the distributed geometric parameter againstacceptable values.
 5. The method of claim 1, further comprising: probingthe plurality of natural eyelashes with a robotic end effector of arobotic mechanism; displacing a first subset of the plurality of naturaleyelashes with the robotic end effector; and after displacing the firstsubset, repeating the measurement of the geometric parameter.
 6. Themethod of claim 5, wherein the robotic end effector comprises a firstprobe, the robotic mechanism further includes a second probe, and therobotic mechanism is configured to controllably displace the first andsecond probes with respect to one another, the method furthercomprising: adjusting a displacement between the first and second probesto be on the same scale as a diameter of an eyelash; plunging the firstand second probes through the plurality of natural eyelashes; andincreasing the displacement between the first and second probes suchthat a second subset of the plurality of natural eyelashes remainsbetween the first and second probes at a lower density than a remainderof the plurality of natural eyelashes.
 7. The method of claim 6, furthercomprising repeating the steps of adjusting the displacement, plungingthe first and second probes, and increasing the displacement until atleast one eyelash is determined to be suitable for extension.
 8. Themethod of claim 5, wherein the robotic end effector comprises a firstprobe, and the robotic mechanism further includes a second probe, themethod further comprising: displacing the first probe through theplurality of natural eyelashes; translating the first probe along theplurality of natural eyelashes; displacing the second probe through theplurality of natural eyelashes; and translating the second probe in anopposite direction of the first probe such that a second subset of theplurality of natural eyelashes remains between the first and secondprobes.
 9. The method of claim 1, further comprising: brushing theplurality of natural eyelashes with a robotic end effector of a roboticmechanism; and after brushing the plurality of natural eyelashes,repeating the measurement of the geometric parameter.
 10. The method ofclaim 1 further comprising, prior to measuring the geometric parameter:a) imaging the plurality of natural eyelashes with the at least onecamera; b) computing a spatial position of the plurality of naturaleyelashes; and c) attempting to isolate the natural eyelash with therobotic mechanism.
 11. The method of claim 10, wherein the at least onecamera includes two cameras.
 12. The method of claim 1, wherein:measuring the geometric parameter includes imaging the natural eyelashwith the at least one camera to create an image; and comparing thegeometric parameter against the acceptable value includes submitting theimage to a neural network.
 13. The method of claim 12, wherein theneural network is trained using a plurality of images of natural humaneyelashes.
 14. The method of claim 13, wherein the plurality of imagesincludes images of singleton eyelashes, crossed eyelashes, and stackedeyelashes.
 15. The method of claim 12, further comprising: measuring ageometric parameter of the natural eyelash with the at least one camera;comparing the geometric parameter against an acceptable value; anddetermining whether the natural eyelash is suitable for extension basedon the comparison.
 16. The method of claim 12, wherein the neuralnetwork is trained using a plurality of images created by a deepgenerator network.
 17. A device configured to determine whether anatural eyelash of a plurality of natural eyelashes of a subject issuitable for extension, the device comprising: a computer vision systemconfigured to observe the plurality of natural eyelashes; a roboticmechanism configured to probe the plurality of natural eyelashes; and acontroller configured to communicate with the robotic mechanism, whereinthe device is configured to: a) image the plurality of natural eyelasheswith the computer vision system; b) compute a spatial position of theplurality of natural eyelashes; c) attempt to isolate the naturaleyelash with the robotic mechanism; d) image the result of the attemptwith the computer vision system; e) when the attempt is successful,determine that the natural eyelash is suitable for extension; and f)when the attempt is unsuccessful, repeat at least steps c and d.
 18. Thedevice of claim 17, wherein the device is further configured to: measurea parameter of the natural eyelash with the computer vision system;compare the parameter against an acceptable value; determine, based onthe comparison, whether the natural eyelash is comprised of multiplenatural eyelashes; and when the natural eyelash is comprised of multiplenatural eyelashes, determine that the natural eyelash is not suitablefor extension.
 19. The device of claim 17, wherein the computer visionsystem includes at least two cameras.
 20. The device of claim 17,wherein the device is further configured to: measure a geometricparameter of the natural eyelash with the computer vision system;compare the geometric parameter against an acceptable value; and verifyisolation of the natural eyelash based on the comparison.
 21. The deviceof claim 20, wherein the computer vision system is a camera, said devicefurther comprising a neural network to which the result is submitted forcomparison to an acceptable value.
 22. The device of claim 17, whereinthe computer vision system is configured to measure a parameter of thenatural eyelash at multiple points along the natural eyelash to generatea distributed parameter, and the device is further configured to:compare the distributed parameter against acceptable values; anddetermine whether isolation has occurred based on the comparison. 23.The device of claim 17, wherein the robotic mechanism includes a firstprobe and a second probe, and the robotic mechanism is configured to:displace the first probe through the plurality of natural eyelashes;translate the first probe along the plurality of natural eyelashes;displace the second probe through the plurality of natural eyelashes;and translate the second probe in an opposite direction of the firstprobe such that a subset of the plurality of natural eyelashes remainsbetween the first and second probes.